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Abstract 

This paper presents We-Share, a social annotation application that enables educators to publish and 
retrieve information about educational ICT tools. As a distinctive characteristic, We-Share provides 
educators data about educational tools already available on the Web of Data while allowing them to 
enrich such data with their experience using technology in the classroom. We-Share evaluation entails 
an empirical study where 23 educators enriched tool descriptions available on the Web of Data out of 
their own experience. The results suggest that experiential annotations published by educators using 
We-Share improve the satisfaction and confidence of other educators when discovering and selecting 
ICT tools. Further, most educators found We-Share an easy-to-use application suitable to share and 
retrieve information about educational ICT tools. 

Keywords: social annotation, collective knowledge systems, Web of Data, Social Web, educational ICT 
tools 


Introduction 

One of the problems that educators need to face when designing a technology-enhanced learning 
situation is the selection of Information and Communication Technologies (ICTs) tools (“tools" 
hereinafter) to support it. This is a critical decision (Gomez-Sanchez, Bote-Lorenzo, Jorrin-Abellan, 
Asensio-Perez, & Dimitriadis, 2009) that requires them to be informed about not only the 
functionality of the tools available or their technological features, but also about their suitability to the 
contexts (Kurti, Spikol, & Millard, 2008) and the pedagogical intentions of the learning situation. As 
several educators typically employ search systems in order to be informed about the tools available 
(Conole, 2008; Madden, Ford, & Miller, 2005), several tool search systems have been proposed for 
them to find this information (e.g., Cool Tools For Schools). The precision of these search system is 
higher than general-purpose ones (e.g., Google) because they focus on a specific domain; however, 
they suffer from common data sustainability problems that limit their utility (Ruiz-Calleja, Vega- 
Gorgojo, Asensio-Perez, Bote-Lorenzo, Gomez-Sanchez, & Alario-Hoyos, 2012): the isolation of their 
datasets, which increases the overall effort required to sustain them; and the well-known cold-start 
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problem (Maltz & Ehrlich, 1995), which hinders the collection of data from the users in socially- 
constructed registries. In order to overcome these problems we developed SEEK-AT-WD (Ruiz- 
Calleja, Vega-Gorgojo, Asensio-Perez, Bote-Lorenzo, Gomez-Sanchez, & Alario-Hoyos, 2014), an 
infrastructure that exploits the data available on the Web to automatically create—and periodically 
update—its tool dataset. 

SEEK-AT-WD currently exposes more than 7000 different tool descriptions that could be exploited by 
educational search systems. These descriptions include functional and technical characteristics of the 
tools. However, existing data about tools in the Web of Data do not include information about how 
tools are employed in educational contexts, which hinders the support provided by this data to the 
educators. As an example, an educator may discover several concept map tools out of the data already 
published on the Web, but it will be hard for her to select an appropriate one for her classroom unless 
she knows which ones have already been used in similar educational contexts. SEEK-AT-WD is 
technically prepared to provide this education-specific information (Ruiz-Calleja et al., 2014) but how 
to collect it remains a problem. A possible solution is to combine the Social Web and the Web of Data 
advantages in a social-semantic approach (Mikroyannidis, 2007). Such an approach involves the 
federation of a tool dataset to the Web of Data while supporting the community of educators when 
enriching or curating the dataset. Thus, the resulting dataset will be automatically updated from non- 
educational data sources and, at the same time, it will contain education-specific information 
published by its community of users. 

How to enable the community of educators to contribute to the sustainment of the dataset publishing 
educational-specific information is key for this social-semantic approach. With this aim, the present 
paper proposes We-Share, a social annotation application that relies on SEEK-AT-WD to support 
educators to publish, enrich, and consume semantic descriptions of tools. We-Share offers a 
convenient and easy-to-use web interface that allows educators to annotate and search tools for their 
needs. Further, this paper provides empirical evidences of the usefulness of the social information 
published with We-Share. Specifically, it describes a case study where 23 higher education 
practitioners from multiple disciplines published information on the Web of Data using We-Share and 
use it later on to discover and select tools. Thus, this proposal assesses the feasibility of enriching data 
from the Web with social educational information and also motivates the use of third-party data from 
the Web for educational purposes. 

The rest of the article is structured as follows: the next section depicts current social and semantic 
approaches followed to sustain registries of educational tools. Then, the following section presents the 
design and development of We-Share. The paper continues by including the description of the 
empirical study and the discussion of its results. Finally, the most important conclusions are 
presented in the last section. 
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Social and Semantic Approaches for the Discovery of Educational 

ICT Tools 

Crawling the Web of Data to Gather Educational Tool Descriptions with SEEK-AT- 
WD 

The Linked Open Data movement promotes the publication of data on the Web with an open license 
and following the Linked Data principles (Berners-Lee, 2006). These principles define a set of best 
practices for data publication that facilitate the consumption and integration of datasets. Thousands 
of data providers, including educational ones (d'Aquin, 2016), are linking their datasets on the Web of 
Data and third parties are reusing this data to build applications. 

Some registries of educational tools publish their content on the Web with an open license, such as 
Cool Tools For Schools or Moodle Plugins Directoiy. However, reusing their data requires a big effort 
because they use their own data formats and schemas. This fact hinders the data integration and 
makes them suffer data sustainability problems, as all their data should be publish by their 
community of users, thus increasing the overall effort of getting tool descriptions. Further, they suffer 
from the so-called “cold-start” problem since, at the beginning, these registries include very few tool 
descriptions, which refrains the community of educators from spending their time publishing 
information in them. In this regard, a step forward has been taken by the ROLE Widget Store and 
Ontoolsearch (Vega-Gorgojo, Bote-Lorenzo, Asensio-Perez, Gomez-Sanchez, Dimitriadis, & Jorrin- 
Abellan, 2010), which publish the functional and administrative characteristics of several tools as 
Linked Open Data. Even though, as they are unable to import data from third-party registries, the 
same sustainability problems still remained. 

Reusing existing Linked Data can help avoid the cold-start problem and the isolation of datasets 
(Ruiz-Calleja et al., 2012). This is the approach followed by SEEK-AT-WD (Ruiz-Calleja et al., 2014), 
which uses the descriptions of tools available on the Web of Data to create, and periodically update, a 
dataset of educational ICT tools. Once a week a crawler extracts tool descriptions from the Web of 
Data and relates them to an education-specific vocabulary, called the SEEK Ontology (Ruiz-Calleja et 
al., 2014). This ontology defines a vocabulary that can be used to describe tools, reviews related to 
these tools, and educational contexts. By January 2016, SEEK-AT-WD retrieved more than 7000 
different tool descriptions, which, once related to the SEEK Ontology using ontology mapping 
techniques (Choi, Song, & Han, 2006), are offered back on the Web of Data to be used by third-party 
applications (note that not all these descriptions were available on the evaluation presented in this 
paper since new tool descriptions appeared on the Web of Data since then). 

SEEK-AT-WD currently offers its data through a browsing interface and a querying interface that 
follow the Linked Data principles. As these interfaces are not suitable for end users, a search system is 
required to enable educators to construct and submit queries to SEEK-AT-WD, as well as to visualize 
the results obtained. Another limitation of SEEK-AT-WD is the current lack of data published on the 
Web about the educational use of the tools, which is specially relevant for the appropriate selection of 
tools by educators (Gomez-Sanchez et al., 2006) (Kurti, Spikol, & Millard, 2008). Specifically, the 
SEEK Ontology includes a set of concepts to describe educational reviews of the tools and the 
educational contexts where these tools were employed (Ruiz-Calleja et al., 2014). Even if no data 
related to these concepts can be currently obtained from the Web, the possibility of semantically 
describing educational contexts is a very interesting feature of SEEK-AT-WD, since they could 
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eventually be exploited when selecting tools and when reviewing them (Gomez-Sanchez et al., 2006; 
Kurti, Spikol, & Millard, 2008). This paper explores the possibility of enabling educators to publish 
this kind of information on SEEK-AT-WD and, as a consequence, on the Web of Data, following a 
social-semantic approach (Mikroyannidis, 2007). 

Social Annotation of Educational ICT Tools for the Web of Data 

The Social Web (O’Reilly, 2007) is a web paradigm in which users move from a purely information¬ 
consuming role to be information publishers. One of its best achievements is to collect and aggregate 
knowledge into community- and domain-specific sites (Gruber, 2008; O’Reilly, 2007), which are 
commonly called “collective knowledge systems" (Gruber, 2008). Several examples can be found 
where collective knowledge systems are employed to recommend tools to educators. They all have to 
face the problem of who and how describes the educational use of the tools so that such information 
can be incorporated in their datasets to facilitate the discovery and selection of tools. Some tool 
registries, such as Schoolforge, separate the users that publish tool descriptions (e.g., tool providers) 
from those that consume them (e.g., educators). Tool providers are expected to contribute since they 
are interested in advertising their tools and they deeply understand their functionality and their 
technical characteristics. However, tool providers cannot always anticipate the educational 
affordances of their tools. In other tool registries, such as Cool Tools For Schools, the community of 
educators is not only expected to obtain tool descriptions, but also to publish descriptions of the tools 
they use in their classrooms. Finally, there are other registries (e.g., ROLE Widget Store) that support 
both roles in contributing to the data publication, thus enabling tool providers to publish their tools 
and educators to enrich the tool descriptions with education-specific information. 

However, all these socially-constructed tool registries are isolated data silos and, hence, they suffer 
from the abovementioned sustainability problems. Furthermore, the educators can only review tools 
by stating a numerical rating or writing an unstructured text. None of these socially-constructed 
registries include a set of concepts to explicitly state the characteristics of the educational use of the 
tools, as SEEK-AT-WD does with the inclusion of the educational contexts in its vocabulary. This lack 
of structure in the reviews simplifies their publication but has some well-known disadvantages 
(Mikroyannidis, 2007). For instance, they lack a coherent categorization scheme (e.g., it is not 
possible to list all the reviews that refer to blended scenarios) and their concepts cannot be used to 
express formal queries (e.g., it is not possible to submit queries such as “I need a voting tool that has 
been employed to support a face-to-face debate"). 

The idea of merging social annotation with semantic descriptions has already been explored in the 
educational domain (Jovanovic, Gasevic, Torniai, Bateman, & Hatala, 2009), although this kind of 
applications typically behave as isolated data silos. Some examples of applications that combine data 
from the Web with information published by their community of users can be found in other domains. 
For instance, Revyu (Heath & Motta, 2007) is a simple application that enables its users to publish on 
the Web of Data reviews of any entity. Other example is presented by Heitmann and Hayes (2010), 
who proposes to use data from the Web to mitigate the data acquisition problems of current 
recommender systems in a similar way as SEEK-AT-WD does. This kind of systems can be accessed 
through an application that hides the complexity of publishing semantically-structured Linked Data to 
their users. In the case of SEEK-AT-WD, such an application is required to enable educators to 
socially publish and curate semantic descriptions of tools as well as to review them. 
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We-Share 

We-Share is a social-semantic annotation application that enables educators to publish information 
about tools on the Web of Data using the SEEK-AT-WD infrastructure (see Figure 1). We-Share 
supports the publication of tools, educational reviews, and educational contexts (as described by the 
SEEK Ontology), as well as updating the already existing ones. This data publication entails the 
management of user accounts to state the authorship of the information published on We-Share and 
the edition rights of each user. Further, We-Share enables educators to retrieve (either browsing or 
querying) and visualize the data published on SEEK-AT-WD in a single environment, as 
recommended for social annotation applications (Neuwirth, 1990). We-Share also offers a usable user 
interface that hides the complexity of managing semantic data since educators are not required to 
have technical skills. 
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Figure 1. We-Share as a social-semantic annotation application that enables educators to publish and 
retrieve information about tools from the Web of Data. 

Logical Architecture of We-Share 

The logical architecture of We-Share consists of the set of components shown in Figure 2, which are 
arranged in a three-tier architecture. This architecture explicitly separates the elements that conform 
the user interface, the presentation tier, from those that implement the processes needed for the 
system to provide the required functionality, the business logic tier, and those that persist the data 
generated and consumed by these processes the data tier. It is also noteworthy that all the 
information about tools managed by We-Share is stored in SEEK-AT-WD so as to be publicly available 
on the Web. Nonetheless, the information related to We-Share users is stored in a separate dataset 
since their privacy should be guaranteed. 
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Figure 2. Logical architecture of We-Share. 

The separation of concerns stressed by the proposed architecture enables the modification of the 
visual appearance of the user interface without affecting the other components of the system. We- 
Share is a data annotation and consuming application (see Figure 1) that directly uses SEEK-AT-WD 
data interfaces without requiring to modify the infrastructure. As a consequence, the Query Manager 
and the Browse Manager not only retrieves data from SEEK-AT-WD, but also parse it to be presented 
by the presentation tier. Similarly, the Data Storing Module aligns the data published by the users to 
the SEEK Ontology before submitting this data to the infrastructure. 

We-Share user Interface Design 

The design of the user interface of the system is specially challenging since it should facilitate 
educators to interact with tool descriptions while hiding the complexity of their computational 
representation. Four different user interfaces need to be designed for We-Share, as shown in Figure 2. 

The Query Interface of We-Share is based on a former search system called Ontoolsearch (Vega- 
Gorgojo et al., 2010), which proposes a direct-manipulation interface to guide educators in the 
construction of formal queries. It includes a graph-based interface that depicts the hierarchy of 
concepts defined by the ontology, thus enabling educators to browse these hierarchies and select the 
concepts that will be used to filter results. In a similar way, the Query Interface depicts the hierarchies 
of tools, educational tasks and artifacts defined by the SEEK Ontology. Additionally, this interface 
includes two forms that enable educators to filter the tool descriptions using the characteristics of the 
educational contexts where the tools have been employed, the technical characteristics of the tools, or 
the keywords contained in their description. The descriptions obtained are presented to the educators 
as a list of tool names, listing first those that have been reviewed. As an example, Figure 3 shows part 
of the tool graph included in the search interface. Educators can select concepts from this graph, add 
them to their query, and use them to filter results. 

The Browse Interface enables educators to browse SEEK-AT-WD knowledge base by representing the 
descriptions of its tools and educational contexts. It also enables users to access collections of entities, 
such as the list of tools that belong to a specific tool type or the ones that support a specific 
educational task. As an example, Figure 4 shows part of the description of Wikispaces as represented 
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by We-Share. This description includes data that was extracted by SEEK-AT-WD from the Web of 
Data and the educational reviews published by educators. Each review includes a textual description, 
and the main characteristics of the educational context where the tool was employed. It is also 
noteworthy that We-Share offers information that was initially obtained from diverse non-educational 
external sources of the Web of Data and educational-specific information published by educators. 


C Si® © seek.cloud.gsic.tel.uva.es 


© 


Video tutorials 
Information about SEEK 


Contribute 

I Add new tool 

| Tool types | Supported tasks 


SEpK 

Information management tool 


1 Add new educational context 
| My tools 


Browse 

List tool types 

ayer 

\ 


Group tool 

List educational contexts 




J Site activity 


-_ 

ModDing tool 

Get some help 


• 



Compeer 



Programming tool 


Questionnaire editor 


V 

Electronic calendar 


task type: Writing Q 
tool type: Group tool Q 
Delivery mode in review: Blended Q 
Teaching technique in review: Peer review Q 


s§IK 


Figure 3. Snapshot of the search interface of We-Share. 

The Data Publication Interface is based on forms, which are commonly employed in social 
applications to facilitate end-users the submission of information to their knowledge base. Three 
different forms can be found in We-Share, which enable the publication and modification of tools, 
educational contexts, and educational reviews. Each form includes a set of text fields where the 
educator can indicate the different parameters of the entity she is describing. The data submitted by 
the educator is then related to the SEEK Ontology and submitted to SEEK-AT-WD by the Data 
Storing Module. Thus, educators can publish complex descriptions of tools, educational reviews, and 
educational contexts manipulating simple forms. 

Finally, the User Management Interface includes a simple interface to log in and out of the system 
(see the left side of Figure 4) and another form that allows educators to manage their accounts or to 
create new ones. 


253 

















Enriching the Web of Data with Educational Information Using We-Share 
Ruiz-Calleja, Asensio-Perez, Vega-Gorgojo, Gomez-Sanchez, Bote-Lorenzo, and Alario-Hoyos 


4 - ■> C :: : 0 seek.doud.gsic.tel.uva.es/weshare/tool/wikispaces ir V (?) 


Search Wikispaces 

Search for tools 


Contribute 

Add new tool 

Add new educational context 
My tools 

My educational contexts 


Browse 

List tool types 

List educational contexts 


DESCRIPTION 

Wikispaces is a Free web hosting service (sometimes called a wiki farm) based in San Francisco, California. Launched in March 2005, Wikispaces 
is owned by Tangient LLC and is among the largest wiki hosts, competing with PBworks, Wetpaint, Wikia, and Google Sites (formerly JotSpot). 
Private wikis with advanced features for businesses, non-profits and educators are available for an annual fee. As of March 2008, they had 
more than 920,000 registered members and hosted more than 390,000 wikis. By March 2009, that number had increased to over 2.2 million 
registered members and more than 900,000 wikis. Wikispaces has also given away more than 100,000 premium wikis to K-12 educators. Since 
2010 Wikispaces have cooperated with web 2.0 education platform Glogster EDU. Glogster EDU embeds Glogs into Wikispaces services. 
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Figure 4. Snapshot of the browsing interface of We-Share showing part of the description of 
Wikispaces (the tasks supported and the artifacts managed by the tool were removed from it). 

Prototype Implementation 

We developed a prototype of We-Share as a Drupal-based web application, which is currently 
available at http://seek.cloud.gsic.tel.uva.es/weshare/ . The development effort was highly reduced 
using Drupal since it includes user and data management facilities that can be directly taken, or easily 
adapted, to We-Share needs. Furthermore, Drupal natively supports the publication of RDF data and 
its architecture follows the same three-tier architecture as We-Share does. Technical details about We- 
Share development can be found at (Ruiz-Calleja, Vega-Gorgojo, Asensio-Perez, Bote-Lorenzo, 
Gomez-Sanchez, & Alario-Hoyos, 2013). 

Different versions of the We-Share prototype were tested by several educators. They used We-Share to 
search and publish tool descriptions providing useful feedback that helped to improve its interface. 
The data published by these tester educators was available during the evaluation of We-Share, even if 
none of them took part in this evaluation. 
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Evaluation 

The aim of this evaluation is to assess whether it is possible for We-Share to gather useful information 
from the community of educator for SEEK-AT-WD data sustainment. The evaluation follows the 
CSCL-EREM framework (Jorrin-Abellan & Stake, 2009) to guide the collection of evaluation data, its 
analysis, and its interpretation. CSCL-EREM distinguishes from other evaluation frameworks since it 
is evaluand-oriented, meaning that it guides researchers to focus on what is being evaluated. It highly 
encourages researchers to gather and analyze data about the evaluand using a mixed methods 
approach (Tashakkori & Teddlie, 2003), which combines the gathering and analysis of both 
qualitative and quantitative data, and is especially suitable to understand social phenomena. 
Specifically, this evaluation adapts the mixed evaluation method described in (Martinez, Dimitriadis, 
Rubia, Gomez-Sanchez, & de la Fuente, 2003). This method combines quantitative and qualitative 
data gathering techniques and triangulates the data gathered to comparatively analyze them in order 
to reach the global conclusions of the evaluation. Thus, the evaluation process is enriched as multiple 
perspectives about the same evaluand are obtained. 

We also followed the anticipated data reduction procedure proposed by (Huberman & Miles, 1994) in 
order to conceptually organize the data gathered. This data was arranged according to three topics, 
defined in Table 2. The first one aims to detect the usability problems that may arise due to the fact 
that We-Share reuses information from the Web of Data. The other two topics assess the impact of the 
social information collected by We-Share. This social information illustrates how the tools were used 
in other learning situations pursuing specific pedagogical objectives, and is also used, in conjunction 
to other information collected from the Web of Data, by educators to discover and select tools. Hence, 
we evaluated whether this social information increases the satisfaction of the educators when they 
discover tools and whether it increases their confidence when they select a specific tool for their 
classrooms. 

Twenty-three higher education educators were voluntarily involved in this evaluation. They first 
published information on We-Share. Then, the impact of the information published was assessed 
through a formal comparison of We-Share versus a search system that does not manage social 
information. This comparison is based on the interactive track of the Text REtrieval Conference 
(TREC) workshops (Lagergren & Over, 1998), which are commonly used in the evaluation of 
interactive information retrieval systems. 

Experimental Setup 

Figure 5 graphically represents the design of the proposed experiment. This design involves two 
differentiated phases: a first part, tool publication, where participants used We-Share to publish 
information, and a second part, tool information retrieval, where the same participants searched for 
tools using two search systems. As participants belong to universities from different countries, an 
asynchronous e-mail-mediated experiment was designed, as described next. 

First, the participants received an e-mail, asking them to fill in a questionnaire ( Q-knownTools ) and 
enumerate some of the tools they had educational experience with. Then, each participant received an 
e-mail with a video tutorial about the data annotation in We-Share and the list of the tools she was 
asked to add or to review. From the tools she had experience with, each participant was asked to 
review those that had previously been published on SEEK-AT-WD, and to add a description and a 
review to those that had not been published yet. After using We-Share to publish information, the 
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participants filled in another questionnaire ( Q-publication ) that contained a Likert-Scale that asked 
for their perceived usability of We-Share regarding data publication. 
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Figure 5. Graphical representation of the tasks carried out by four of the participants of the 
evaluation. 


The second part of the experiment included a formal comparison between We-Share search system 
and a non-social search system implemented as a stand-alone application described in (Vega-Gorgojo, 
Ruiz-Calleja, Asensio-Perez, & Jorrin-Abellan, 2012). The latter also collects data from SEEK-AT-WD 
but does not manage the social information included in the infrastructure. Six search tasks (TA-TB- 
TC-TD-TE-TF) were chosen for the study (see Table 1). As the search tasks were intended to be 
representative of real information needs, they were selected from the pool of questions posed by 
educators to evaluate the initial dataset of SEEK-AT-WD (Ruiz-Calleja, et al. 2012). 

Table 1 

Overview of the Search Tasks used in the Experiment 


ID 

Brief description 

TA 

I would like a tool to edit video clips (it would be better if it also allows to edit audio and 
images). This tool has to be free. 

TB 

In an on-line course I ask my students to do a written exercise and then review the work of 
two of their partners. I would like a tool to support this peer review activity. 

TC 

I would like a tool that facilitates meetings between people that are physically in the same 
place and other people who are not. The communication cannot be based on textual 
messages. 

TD 

I am a teacher of a course of engineering. I would like a tool to make questionnaires to my 
students, as well as to ask their opinions. 

TE 

I would like to recommend my students to use LaTeX and, as most of them are Windows 
users, I would like to recommend a LaTeX editor that works on Windows. 

TF 

In a face-to-face laboratory I want my students to create a concept map with the most 
important concepts of the subject. I would like a tool for them to do this task. 


256 















Enriching the Web of Data with Educational Information Using We-Share 
Ruiz-Calleja, Asensio-Perez, Vega-Gorgojo, Gomez-Sanchez, Bote-Lorenzo, and Alario-Hoyos 


When the second part of the experiment began, each participant received an e-mail with a video 
tutorial that explained how to search tools with both systems and four information needs that they 
were asked to satisfy. The search tasks assigned to each participant and the performance order were 
randomly selected. The participants were asked to search for tools that would satisfy the information 
need and to select the one they would employ in the given situation. They were asked to fill in a 
questionnaire ( Q-search ) for each information need where they reported their satisfaction on the tools 
discovered and their confidence on the tool selected in a 1 to to scale (these were the metrics used to 
compare the performance of both search systems), the tool they chose, the reason why they chose that 
tool, and any other observation they wanted to state. When they finished all the search tasks they 
completed another questionnaire ( Q-exit ) where their global opinion about We-Share utility and 
usability was reported using a Likert-Scale. 

The satisfaction on the tools discovered and the confidence on the tools selected were computed for 
each observation. The factors that were considered to influence the satisfaction on the tools 
discovered were the search system (the primary factor) and the search task. On the other hand, the 
confidence on the tool selected was considered to be affected by the search system (the primary 
factor), the search task and the satisfaction on the tools discovered. A quantitative analysis of the 
results was carried out, which included an Analysis of Variance (ANOVA) model, thus addressing the 
question of whether these two metrics were significantly different using both search systems. 
Furthermore, a deeper comprehension of the search processes was obtained by analyzing We-Share 
logs and the opinions of the participants collected from the different questionnaires. 

Table 2 

Summary of Evaluation Topics, Data Sources, and Data Processing Techniques 


Evaluation topic 

Data sources 

Data processing techniques 

Does We-Share present usability problems 
due to the origin of its data that hinder 
educators to publish information? 

Q-publication 
We-Share logs 

Descriptive statistics 

Does the social information gathered by We- 
Share increase the satisfaction of educators 
when discovering tools? 

Q-Search 

Q-exit 

Application logs 

ANOVA 

Descriptive statistics 

Does the social information gathered by We- 
Share increase the confidence of educators 
when selecting a specific tool? 

Q-Search 

Q-exit 

ANOVA 

Descriptive statistics 


In this study, 23 educators from 12 different universities and 4 different countries participated as non- 
rewarded volunteers, although 2 of them did not completed the second part. The participants did not 
have any kind of relationship among themselves and most of them did not know each other. They had 
teaching experience in Computer Science (10), Telecommunications Engineering (9), Pedagogy (3), 
and Psychology (1). They all had experience using Social Web applications and employing technology 
for educational purposes. 

Findings on the Data Publication Process 

The participants published 23 tool descriptions, 39 educational contexts, and 80 educational reviews. 
All this data was combined with the data available in SEEK-AT-WD, which had been previously 
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obtained from the Web of Data or published by the tester educators who provided feedback about We- 
Share interface, as Table 3 summarizes. 

Table 3 

Data Available in SEEK-AT-WD Before the Second Part of the Evaluation Started 


Provided by 

Tool descriptions 

Educational 

contexts 

Educational 

reviews 

Evaluation participants 

23 

39 

80 

Tester educators 

26 

29 

96 

Obtained from the Web of Data 

6760 

0 

0 

Total 

6809 

68 

176 


Table 4 

Results of the Questionnaire Related to We-Share Publication Usability 



Strongly 

disagree 

Disagree 

Somewhat 

disagree 

Somewhat 

agree 

Agree 

Strongly 

agree 

NA/NK 

I find it to be 
easy to use 

0 

1 

2 

1 

11 

6 

2 

I understand 
the meaning 
of its fields 

0 

0 

0 

3 

12 

8 

0 

I find easy to 
interact with 
its menus 

0 

1 

1 

1 

11 

6 

2 

I find easy to 
publish tools 

1 

0 

1 

5 

6 

8 

2 

I find easy to 
review tools 

0 

1 

0 

2 

6 

14 

0 


The participants' opinion about the data publication was collected and the information they published 
was analyzed. Table 4 collects the results of the Likert-scale questionnaire regarding the data 
publication in We-Share ( Q-publication ). It can be seen that most of the participants agreed or 
completely agreed that We-Share is an easy-to-use annotation application. Most of them understood 
the meaning of its menus and they found it easy to interact with them. Additionally, the publication of 
tool descriptions and reviews in We-Share was not perceived as a difficult task by most of the 
participants. In fact, they properly published tool descriptions, reviews, and educational contexts, and 
all the educators made a correct use of We-Share. 

Most of the comments posed by the participants regarding the data publication were positive. A few 
participants complained that We-Share requires to understand the SEEK Ontology in order to 
properly classify tools and relate them to educational contexts. This “cognitive cost" (Mathes, 2004) is 
a well-known problem that appears when users relate the information they publish to a given 
vocabulary, since they have to do the effort of understanding such vocabulary. Despite this, 
participants socially agreed on the meaning of these concepts by reading some examples of tools and 
educational contexts that had already been published in We-Share. As the same participants that 
complained about this aspect, agreed or strongly agreed that We-Share is a usable application, it can 
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be considered that they understood this cognitive cost as part of the learning process required to 
annotate tool descriptions with We-Share. 

Findings on the Tool Discovery Process 

During the second part of the evaluation, a total amount of 84 tool discovery processes were carried 
out. In 82 of them, the participants could find useful tools to satisfy the search tasks requested. The 
two discovery processes that could not be completed corresponded to the first time two different 
participants used the We-Share search system, so they may be due to the learning process required to 
get used to We-Share. Moreover, only in four observations, the satisfaction when discovering tools 
reported was lower than 5 out of 10. 

Table 5 

Arithmetic Mean (from o to 10) of the Satisfaction When Discovering Tools Reported by Participants 
for We-Share and the Non-Social Search System 



TA 

TB 

TC 

TD 

TE 

TF 

Total 

We-Share 

5-86 

6.63 

7-65 

7.14 

8.50 

7-50 

7.17 

Non-Social Search System 

5-42 

6-33 

7.14 

7-33 

7-35 

7.86 

6.76 

Total 

5.62 

6.50 

7.41 

7-23 

7.84 

7.69 

6.97 


An ANOVA model was fitted for the satisfaction with the factors considered in the design. A valid 
model was obtained and it showed that the difference of the participants' satisfaction using both 
search systems was not statistically significant. In order to further understand the impact of the social 
information on the participants' satisfaction about the tools discovered, the mean of the satisfaction 
was calculated for each of the search tasks, as shown in Table 5. It can be seen that when the social 
information is available the satisfaction is higher, although only slightly better results are obtained. 

The queries submitted by the participants were also analyzed. 255 queries were submitted for 42 
search tasks without social information and another 256 for 42 search tasks with social information. 
The quantity of search tasks required to satisfy the information needs and the type of filters used were 
very similar in both search systems. In fact, when the social information was available only 25.1% of 
the queries submitted used concepts related to educational contexts to filter results. As most of the 
queries submitted by the participants with both search systems used the same filters, in most cases the 
results obtained were the same. This fact explains why the satisfaction on the tools discovered is so 
similar. 

Table 6 

Results of the Questionnaire Related to the Usability and Utility of We-Share when Supporting the 
Tool Discovery and Selection 



Strongly 

disagree 

Disagree 

Somewhat 

disagree 

Somewhat 

agree 

Agree 

Strongly 

agree 

NA/NK 

I find it to obtain 
tool descriptions 
using We-Share 

0 

1 

2 

4 

9 

5 

0 

I find its search 
system to be easy 

0 

1 

0 

3 

9 

8 

0 
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to use 








Using it the time 
spent to discover 
and select tools is 
reasonable 

0 

1 

0 

2 

12 

6 

0 

The data 
published by the 
community of 
educators is 
apparent to me 

0 

1 

0 

2 

12 

3 

3 

I find We-Share to 
be useful in my 
job 

0 

0 

0 

6 

9 

6 

0 


In questionnaire Q-exit the participants were also asked about their opinion of the usability of the 
We-Share search system (see Table 6). As a general rule, the participants found it easy to search for 
tools using We-Share. Specifically, 14 out of 21 participants considered easy to obtain tool descriptions 
using We-Share and 18 out of 21 found it an easy-to-use search system. Additionally, some 
participants made comments about the usability of the search system. Some of them explicitly said 
that they liked the interface. Nonetheless, a few participants pointed out two well-known usability 
problems: their need to understand the SEEK Ontology and the difficulty to translate a search task 
into a boolean expression with precise semantics. These are two well-known problems in information 
retrieval systems that manage structured data that were also detected in other search systems (e.g., 
Vega-Gorgojo et al., 2010). 

Findings on the Tool Selection Process 

During the second part of the evaluation process the participants also reported their confidence on the 
tools they selected on each search task. An ANOVA model was fitted with the participants' confidence 
on the tools selected and the factors considered in the design. Then, the ANOVA assumptions were 
checked in order to assess its validity (Jain, 1991). For a 90% confidence interval the difference of the 
confidence of the tools selected with We-Share with respect to the non-social search system is in the 
interval (0.0283,1.4053) with mean 0.7168. This difference shows that the search system employed 
has a statistically significant influence on the confidence of the tools selected by the participants, 
having better results We-Share. 

Table 7 

Categorization of the Arguments Provided by the Participants in Questionnaire Q-Search to Justify 
Their Tool Selection. 



Function 

Previous 

experience 

Technical or 
administrative 

Reviews 

Educational 

contexts 

Other 

We-Share 

37% 

22% 

12% 

29% 

20% 

24% 

Non-Social Search 
System 

41% 

27% 

17% 

— 

— 

27% 

Total 

39% 

24% 

15% 

15% 

10% 

26% 


Note. Each tool selection can be based on several types of arguments. 


Table 7 shows the type of arguments reported by the participants when selecting tools using both 
versions of the search system (note that each tool selection can be based on several types of 
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arguments). It can be seen that when the social information was not available, the functionality of the 
tools, and the previous experience of the participants were the two main arguments for their selection. 
On the other hand, when the social information was available, these two arguments were also very 
frequent, but many educators also based their selections on the reviews and the educational contexts 
related to the tool descriptions. Interestingly, when the tool selection was based on the experience of 
other educators, the reported confidence was usually higher. In fact, only one of them had a 
confidence lower than 7 out of 10, and their arithmetic mean (8.20/10) was higher than the mean of 
the confidence in the rest of the experiments (7.24/10). 

Table 8 

Number of Tools that had Reviews when Selected 



TA 

TB 

TC 

TD 

TE 

TF 

Total 

We-Share 

1/6 

8/8 

7/8 

7/7 

5/6 

4/6 

32/41 

Non-Social Search System 

0/7 

3/6 

4/7 

6/6 

2/7 

5/8 

19/41 

Total 

1/13 

11/14 

11/15 

13/13 

7/1.3 

9/14 

51/82 


There were also differences on the tools selected by the participants when the social information was 
available. Table 8 shows, for each search task, the fraction of the tools selected that had reviews 
attached to them (note that when using the non-social search system the participants did not know 
which tools had reviews). When the social information was not available, educators selected tools with 
reviews in 19 out of 41 cases, while these tools were selected in 32 out of 41 cases when the social 
information was available (68.4% of increase). 

Q-exit asked participants about their perception of the data added by the educational community 
using We-Share. Fifteen out of 18 participants agreed or completely agreed that the data published by 
the community of educators is apparent to them. Additionally, this questionnaire asked participants 
for their perceived utility about We-Share. Fifteen out of 21 participants found We-Share a useful tool 
in their job, while 18 out of 21 participants agreed or completely agreed that the time spent to discover 
and select tools using We-Share is reasonable. 

Discussion of the Evaluation Results 

The evaluation entailed a group of educators who used We-Share to publish, enrich, and retrieve tool 
descriptions from the Web of Data. Some information related to the usability of We-Share was 
collected, showing that the participants did not find special problems to annotate tool descriptions, 
nor for coherently combine the information they publish with the tool descriptions already available 
on the Web. Further, the problems detected when participants searched tools using We-Share were 
very similar to those detected in other evaluations where end users submit semantic and boolean 
queries to a manually-created dataset. These are remarkable findings since they show that tool 
descriptions from the Web of Data can be combined with social educational-specific information to 
support educators when discovering and selecting tools. Thus, We-Share can exploit the sustainability 
advantages of reusing updated data from external sources of the Web while it is still able to provide 
educational-specific information about tools. 

Another important conclusion of the evaluation is related to the impact of the social information when 
discovering tools. The reported satisfaction of educators when discovering tools slightly increased 
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when the social information was available. The analysis of the queries submitted by the participants 
showed that the participants did not make extensive use of the parameters that filter tools depending 
on their educational use. For this reason the social information was not fully exploited during the 
experiment. Even though, the social information had an impact on the participants when selecting 
tools. When this social information was available, their confidence when selecting ICT tools 
significantly increased. Further, many participants justified their tool selection based on this social 
information and they tended to select those tools that had been reviewed by other educators. 

All these findings support the conclusion that We-Share is useful to enrich with educational-specific 
information the information about tools that is published on the Web of Data. Nonetheless, the results 
of this evaluation are affected by several aspects. One of the most important ones is the SEEK 
Ontology, which defines the parameters to describe tools and educational contexts and restricts the 
formal queries that can be submitted to SEEK-AT-WD. Another aspect is the data obtained by SEEK- 
AT-WD from the Web of Data, which was gathered from non-educational data sources and it is 
sometimes difficult to relate the data gathered from them to the SEEK Ontology (Ruiz-Calleja et al., 
2014). The evaluation may also be conditioned by the higher-education context and by the fact that 
most of the educators who participated in the happenings had a technical background. The impact of 
these two aspects was not assessed in the study carried out. 


Conclusions and Future Work 

This paper proposed We-Share, an application that allows educators to publish, annotate, and retrieve 
tool descriptions available on the Web of Data by making use of the SEEK-AT-WD infrastructure. A 
working prototype of We-Share was developed and used in an empirical experiment where a group of 
23 educators participated. The evaluation showed that We-Share is perceived as an easy-to-use 
application that is useful to satisfy realistic information needs regarding educational tools. 
Interestingly, it was shown that the social information collected had a positive impact, both when 
participants discovered and selected tools, but this impact was higher in the tool selection process. It 
is also remarkable that no usability problems were detected on We-Share due to the origin of the data 
it manages. 

These results lead us to conclude that We-Share supports the community of educators to take 
advantage of the Web of Data. Rich semantic descriptions of ICT tools can be automatically obtained 
from the Web of Data and further enriched with educational-specific information. Thus, We-Share 
combines updated information about the technical and administrative aspects of the tools with 
experiences of their educational use. In addition, as the data published by We-Share is also available 
as Linked Open Data, it can be further exploited by other educational applications, thus allowing to 
share information among educational communities. The combination of these technological and 
pedagogical perspectives in the information provided by We-Share about the tools is recognized by 
well-known theoretical frameworks, such as TPACK (Koehler, Mishra, Kereluik, Shin, & Graham, 
2014), as one of the crucial aspects to be promoted in Teacher Professional Development (TPD) efforts 
aimed at the effective integration of ICT in education. 

One of the future research lines is indeed the exploration of the capabilities of We-Share in TPD 
programs aimed at training non-experienced teachers in the use of ICT tools for innovative teaching 
situations. In this sense, We-Share is understood as a shared repository to collect the innovative use of 
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ICT tools for learning purposes and to promote the spread of innovations inside a community of 
educators. It would also be very interesting to integrate the social annotations published through We- 
Share to other social annotations of educational tools that may appear in the Web of Data. 
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